The new features of AbiPy v0.9.1¶

M. Giantomassi and the AbiPy group¶

10th international ABINIT developer workshop
May 31 - June 4, 2021 - Smart Working, Lockdown

  • These slides have been generated using jupyter, nbconvert and revealjs
  • The notebook can be downloaded from this github repo
  • To install and configure the software, follow these installation instructions

Use the Space key to navigate through all slides.

What is AbiPy?¶

Python package for:¶

  • Generating ABINIT input files automatically
  • Post-processing output results (netcdf and text files)
  • Interfacing ABINIT with external tools
  • Defining and executing ABINIT-specific workflows

Dependencies:¶

  • Hard deps: numpy, scipy, pandas, matplotlib, netcdf4, plotly, sympy, pymatgen
  • Soft: ipython, jupyter, phonopy, ASE.

NB: AbiPy relies on pymatgen but it be interfaced with other tools (e.g ASE, phonopy, etc) via converters

What's new at the level of the installation procedure?¶

  • AbiPy now requires python >= 3.6
  • conda packages for AbiPy and ABINIT are now provided by conda-forge
  • AbiPy v1.0 will (finally) drop support for Abinit8
  • We plan to provide recipes for EasyBuild and Spack
  • We seek for volunteers to support other package managers (homebrew, apt-get, etc)

How to install AbiPy¶

Using conda and the conda forge channel (recommended):

    conda install abipy --channel conda-forge

Recipes for AbiPy and ABINIT are now supported by the conda-forge community 🎉

This means that:

  • conda packages are automatically generated when new ABINIT/AbiPy releases are pushed to github
  • the abiconda channel is deprecated. Please use the conda-forge versions

Since conda is not limited to python packages, one can easily install ABINIT in the same env with:

    conda install abinit -c conda-forge

Using pip and python wheels:

    pip install abipy --user

From the github repository (develop mode):

    git clone https://github.com/abinit/abipy.git
    cd abipy 
    python setup.py develop

For further info see http://abinit.github.io/abipy/installation.html

What's new at the level of the documentation?¶

  • New website based on Read-the-Docs theme
  • Gallery with matplotlib and plotlyt examples
  • New examples for flows
  • Each example can be executed on mybinder
  • Jupyter notebooks
  • abitutorial github repo with additional example

New AbiPy website based on Read-the-Docs theme¶

In [2]:
%embed https://abinit.github.io/abipy/index.html
Out[2]:

Sphinx-gallery with docs, figures and links to download py scripts, notebooks or run the example via binder.¶

In [3]:
%embed https://abinit.github.io/abipy/gallery/plot_phbands_nkpt_tsmear.html#sphx-glr-gallery-plot-phbands-nkpt-tsmear-py
Out[3]:
In [4]:
%embed https://abinit.github.io/abipy/gallery/index.html
Out[4]:
In [5]:
%embed https://abinit.github.io/abipy/flow_gallery/index.html
Out[5]:
In [6]:
%embed https://nbviewer.jupyter.org/github/abinit/abitutorials/blob/master/abitutorials/index.ipynb
Out[6]:

What's new at the level of workflow machinery?¶

For the DFPT part:¶

  • Python converters DDB $\;\rightleftarrows \;$ phonopy / tdep

  • New Flows to compute:

    • elastic and piezoelectric tensors with DFPT (clamped/relaxed atoms)
    • Gruneisen parameters from DFPT calculations
    • Quasi-harmonic approximation with DFPT
    • effective masses with DFPT or finite differences

For the e-ph part:¶

  • New flows and post-processing tools for:

    • Imaginary part of ph-e self-energy
    • e-ph self-energy
    • transport properties witing SERTA/MRTA/IBTE
    • e-ph matrix elements and scattering potentials

Two different workflows infrastructures:¶

Internal implementation (abipy.flowtk Flows):¶

  • ✅ Lightweigth, no database is required
  • ✅ Designed for rapid prototyping and for testing advanced ABINIT capabilities
  • ❌ No explicit support for high-throughput (HT) applications.

AbiFlows package (requires Fireworks and MongoDB database)¶

  • ✅ HT-oriented: use database to store workflow status for further analysis
  • ✅ High-level no-brainer API designed for HT applications
  • ❌ New ABINIT features are first implemented/tested with flowtk Flows and then ported to AbiFlows.

NB: There's an ongoing effort to reimplement AbiFlows in terms of the atomate framework so this talk will mainly discuss the features available in abipy.flowtk.

DDB converters¶

(contributed by G. Petretto)

  • Based on py script by A. Romero. ...

  • Applications:

    • Interface AbiPy with phonopy
    • Connect ABINT DPT with other requiring phonony (e.g. anharmonic calculations hipive)

Example: obtain irreps from DDB using phonopy

ElasticWork¶

This flow computes:¶

  • the rigid-atom elastic tensor
  • the rigid-atom piezoelectric tensor (insulators only)
  • the internal strain tensor
  • the atomic relaxation corrections to the elastic and piezoelectric tensor

For the formalism, see:

Python API:¶

scf_input = make_scf_input() # Build input for GS calculation

elast_work = flowtk.ElasticWork.from_scf_input(scf_input, 
                                               with_relaxed_ion=True, 
                                               with_piezo=True)

To compute elastic tensors from the DDB file, use:¶

In [7]:
elastic_ddb = abilab.abiopen("elastic_DDB")
edata = elastic_ddb.anaget_elastic()

edata.elastic_relaxed
Out[7]:
xx yy zz yz xz xy
Voigt index
xx 135.262182 54.450376 38.052927 0.00000 0.000000 0.000000
yy 54.450376 135.262181 38.052927 0.00000 0.000000 0.000000
zz 38.052927 38.052926 148.211029 0.00000 0.000000 0.000000
yz 0.000000 0.000000 0.000000 30.55071 0.000000 0.000000
xz 0.000000 0.000000 0.000000 0.00000 30.550709 0.000000
xy 0.000000 0.000000 0.000000 0.00000 0.000000 40.405903
In [8]:
edata.get_elastic_properties_dataframe(properties_as_index=True)
Out[8]:
property 0 1
0 trans_v 3.194459e+03 3.838052e+03
1 long_v 5.796035e+03 6.295200e+03
2 snyder_ac 5.767044e+01 8.693459e+01
3 snyder_opt 3.158141e-01 3.621134e-01
4 snyder_total 5.798626e+01 8.729670e+01
5 clarke_thermalcond 7.734051e-01 9.006348e-01
6 cahill_thermalcond 8.539415e-01 9.791319e-01
7 debye_temperature 3.760623e+02 4.477874e+02
8 k_voigt 7.553865e+01 7.553930e+01
9 k_reuss 7.553074e+01 7.553579e+01
10 k_vrh 7.553469e+01 7.553754e+01
11 g_voigt 3.951341e+01 5.767416e+01
12 g_reuss 3.761300e+01 5.366047e+01
13 g_vrh 3.856321e+01 5.566731e+01
14 universal_anisotropy 2.527308e-01 3.740360e-01
15 homogeneous_poisson 2.818554e-01 2.041909e-01
16 y_mod 9.886491e+10 1.340681e+11
In [9]:
edata.get_elastic_voigt_dataframe(tol=1e-5)
Out[9]:
voigt_cinds elastic_relaxed elastic_clamped
0 (0, 0) 135.262182 165.988592
1 (0, 1) 54.450376 40.464803
2 (0, 2) 38.052927 21.090298
3 (0, 3) 0.000000 0.000000
4 (0, 4) 0.000000 0.000000
5 (0, 5) 0.000000 0.000000
6 (1, 0) 54.450376 40.464802
7 (1, 1) 135.262181 165.988592
8 (1, 2) 38.052927 21.090299
9 (1, 3) 0.000000 0.000000
10 (1, 4) 0.000000 0.000000
11 (1, 5) 0.000000 0.000000
12 (2, 0) 38.052927 21.090298
13 (2, 1) 38.052926 21.090298
14 (2, 2) 148.211029 182.585743
15 (2, 3) 0.000000 0.000000
16 (2, 4) 0.000000 0.000000
17 (2, 5) 0.000000 0.000000
18 (3, 0) 0.000000 0.000000
19 (3, 1) 0.000000 0.000000
20 (3, 2) 0.000000 0.000000
21 (3, 3) 30.550710 40.818194
22 (3, 4) 0.000000 0.000000
23 (3, 5) 0.000000 0.000000
24 (4, 0) 0.000000 0.000000
25 (4, 1) 0.000000 0.000000
26 (4, 2) 0.000000 0.000000
27 (4, 3) 0.000000 0.000000
28 (4, 4) 30.550709 40.818195
29 (4, 5) 0.000000 0.000000
30 (5, 0) 0.000000 0.000000
31 (5, 1) 0.000000 0.000000
32 (5, 2) 0.000000 0.000000
33 (5, 3) 0.000000 0.000000
34 (5, 4) 0.000000 0.000000
35 (5, 5) 40.405903 62.761895

For additional info, see the notebook tutorial:

https://nbviewer.jupyter.org/github/abinit/abitutorials/blob/master/abitutorials/elastic/lesson_elastic.ipynb

Gruneisen parameters¶

This flow computes:¶

  • $\gamma_{\bf{q}\nu}$ using finite differences and DFPT phonons

Python API:¶

scf_input = make_scf_input() # Build input for GS calculation

# NB: k-mesh in gs_inp and ngqpt q-mesh must be commensurate.
from abipy.flowtk.gruneisen import GruneisenWork
voldelta = gs_inp.structure.volume * 0.02
work = GruneisenWork.from_gs_input(gs_inp, voldelta, ngqpt=[2, 2, 2], with_becs=False)
  • Three relaxations at fixed volume (violet task)
  • Each relaxation task starts a DFPT computation of the dynamical matrix.

Non-linear optical properties¶

This flow computes:¶

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See J. Abreu's talk for applications

For the formalism see:

Python API:¶

scf_input = make_scf_input()   # Get the SCF input (without SOC)

scf_input = make_scf_input()
return flowtk.NonLinearCoeffFlow.from_scf_input("nlo_flow", scf_input)

Effective masses with DFPT¶

This flow computes:¶

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See J. Abreu's talk for applications

For the formalism see:

Python API:¶

scf_input = make_scf_input()   # Get the SCF input (without SOC)

from abipy.flowtk.effmass_works import EffMassAutoDFPTWork
flow = flowtk.Flow("effmass_flow")

work = EffMassAutoDFPTWork.from_scf_input(scf_input)
flow.register_work(work)
  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See Joao's talk

e-ph matrix elements along an arbitrary q-path¶

This flow computes:¶

This example shows how to compute the e-ph matrix elements in AlAs along a q-path with AbiPy flows. The final results are stored in the GKQ.nc file (one file for q-point) in the outdata of each task.

Python API:¶

# Build input for GS calculation on a 2x2x2 k-mesh
scf_input = make_scf_input(ngkpt=(2, 2, 2))

# q-mesh for phonons
ngqpt = (2, 2, 2)

# Create flow to compute all the independent atomic perturbations
# Use ndivsm = 0 to pass an explicit list of q-points.
# If ndivsm > 0, qpath_list is interpreted as a list of boundaries for the q-path
qpath_list = [[0.0, 0.0, 0.0], [0.01, 0, 0], [0.1, 0, 0],
              [0.24, 0, 0], [0.3, 0, 0], [0.45, 0, 0], [0.5, 0.0, 0.0]]

from abipy.flowtk.eph_flows import GkqPathFlow
flow = GkqPathFlow.from_scf_input("flow_dir", scf_input,
                                  ngqpt, qpath_list, ndivsm=0, with_becs=True,
                                  ddk_tolerance={"tolwfr": 1e-8})

Formalism:

https://abinit.github.io/abipy/flow_gallery/run_gkq.html#sphx-glr-flow-gallery-run-gkq-py

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See Joao's talk

Other applications of AbiPy flows, are discussed in the following talks:¶

  • (G.Brunin)
  • (Gabriel's PhD student
  • Julien

What's new at the level of the post-processing tools?¶

  • New plotting tools based on plotly
  • Post-processing tools for e-ph calculations
  • GUIs and dashboards based on panel and bokeh

Matplotlib vs Plotly¶


  • ✅ Publication quality figures.
  • ✅ Flexible python API able to produce rather advanced plots
  • ❌ Plots are difficult to customize without changing the python code
  • ❌ Plots lacks interactivity and integration with HTML.



  • ✅ Interactive plots + chart editor GUI to customize the figure
  • ✅ Play well with HTML (plotly us written in js with python bindings)
  • ❌ Open source project but not all the features are available in the free plan
  • ❌ Requires browser (this may represent an issue on some HPC centers)

AbiPy plots with matplotlib¶

In [10]:
gsr = abiopen("si_nscf_GSR.nc")
gsr.ebands.plot(with_gaps=True);

AbiPy plots with plotly¶

(contributed by Y. He and MG)

In [11]:
gsr.ebands.plotly(with_gaps=True);  # object.plot becomes object.plotly

To update the plotly figure to the chart studio server, use:¶

gsr.ebands.plotly(with_gaps=True, chart_studio=True);

Now users can finally customize the plot without changing the py code.

Interactive 3d plots with plotly:¶

In [14]:
gsr.ebands.kpoints.plotly();

The matplotlib 3d plot embedded in HTML :¶

In [15]:
gsr.ebands.kpoints.plot();

Important note¶

  • AbiPy will continue to support and develop matplotlib-based tools
  • We plan to implement plotly-based tools for the most important physical properties but we cannot support all the possible options already implemented with matplotlib
  • Fortunately, there are packages such as holoviz or plotly-express that provided the data is stored in a pandas dataframe.

Several other plotting libraries are available in the pydata ecosystem

Integrating Abipy with web-based technologies¶

  • Integration between panel and AbiPy
  • How to create GUIs inside jupyter notebooks
  • Dashboards and web apps
  • Integration with the AbiPy command line interface

What is Panel?¶

  • Panel provides tools for composing widgets, plots, tables into apps and dashboards
  • It relies on the client-server model where:
    • the client is the web browser running HTML/CSS/JS code.
    • the server communicates with the client, executes py code and sends the results back to the client
  • In a nutshell, panel keeps the browser and python in synch e.g. the user clicks a button in the GUI and the signal to python.

Why panel?¶

  • Panel works with visualizations from Bokeh, Plotly, Matplotlib, HoloViews, and many other Python plotting libraries,
  • Panel works equally well in Jupyter Notebooks

Thanks to this infrastructure, developers can

  • use python to solve the scientific problem using the pydata ecosystem and other ab-inition py packages
  • use modern web-technologies to present the results to the client without having to deal with HTML/CSS/JS

Pros and cons of the client-server model¶

Advantages:¶

  • Client does not need to install the scientific software stack (when running on different machines)
  • Can take advantage of modern HTML technologies to present the results in the browser
  • Can implement web apps that allows the user to upload the data and analyze the results withouth having to install python software.

Disavantages¶

  • Round trip delay if client != host and slow connection
  • Upoloadng a 1Gb file to the remote server just because you don't want to install software on the localhost is a very bad idea.
  • OK for relatively small files (< 1Gb) but this approach is not designed to handle big data.
  • Not all the HPC centers provide specialized nodes to post-process the results inside a web browser/notebook.

How to use AbiPy panels inside jupyter notebooks¶

To build a panel GUI inside the notebook, use:

In [16]:
#gsr = abiopen("si_nscf_GSR.nc")
#abilab.abipanel();
#gsr.get_panel()
In [17]:
ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")

abilab.abipanel(); # Important
ddb.get_panel()
Out[17]:
Don't be surprised if you start to click buttons and **nothing happens**! One needs a **running python backend** to execute the callbacks triggerered by the GUI widgets.

Other files are supported as well¶

In [18]:
ddb.structure.get_panel()
Out[18]:

Creating dashboards from the command line¶

  • Using panel inside notebooks is great if you need both the flexibility of the python language and the easiness of the GUIs to improve your productivity.
  • In some cases, however, we just want to create a dahhboard with widgets to interact with the data.

To create a panel dashboard associated to the DDB file, use:

abiopen.py out_DDB --panel  # or -pn if you prefer the short version.

To produce a predefined set of matplotlib figures, use:¶

abiopen.py mgb2_kpath_FATBANDS.nc --expose --seaborn

abiopen.py mgb2_kpath_FATBANDS.nc --expose-web # -ewb

abiopen.py mgb2_kpath_FATBANDS.nc --plotly # -ply

abiopen_expose

Replace --expose with --notebook to generate a jupyter notebook with predefined python code¶

Creating dashboards without widgets with abiview¶

to get a quick look at the results

abiview.py ddb ZnSe_hex_qpt_DDB --panel  # or -pn if you prefer the short version.
In [19]:
## Documentation of new features available at:
In [20]:
%embed https://abinit.github.io/abipy/graphical_interface.html
Out[20]:

Need to call anaddb to compute and visualize ph-bands and DOS from DDB?¶

abiview.py ddb ZnSe_hex_qpt_DDB --seaborn
In [22]:
#plotter = abilab.ElectronBandsPlotter()
#plotter.add_ebands(label="BZ sampling", bands="si_scf_GSR.nc")
#plotter.add_ebands(label="k-path", bands="si_nscf_GSR.nc")
#plotter.gridplot(with_gaps=True);

Bash is handy but python is more flexible¶

Let's use the DdbRobot to compare phonons obtained with different ${\bf k}$-meshes and smearing values:¶
In [28]:
#multi = abilab.ebands_input(structure="si.cif", 
#                            pseudos="14si.pspnc",
#                            ecut=8, 
#                            spin_mode="unpolarized", 
#                            smearing=None, 
#                            dos_kppa=5000)

#multi.get_vars_dataframe("kptopt", "iscf", "ngkpt")

Future developments¶

Post-processing tools¶

  • Implement panel GUIs for all the netcdf files already supported by AbiPy
  • Develop web applications so that users can upload e.g. DDB files and analyze the results

Continuous Integration¶

  • Use AbiPy programmatic interface to implement:

    • Validation of parallel algorithms for np in range(1, N)
    • Stress testing
    • Benchmarks